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Introduction 


Developed by the National Writing Project (NWP), the College, Career, and Community Writers 
Program (C3WP) seeks to improve students’ argument writing by building teachers’ 
understanding of and skill in teaching source-based argument writing.’ The program features 
intensive professional development (PD), a set of instructional resources, and formative 
assessment tools designed to help teachers bring C3WP into their classrooms. 


Under a 2012 Investing in Innovation (i3) Validation grant from the U.S. Department of 
Education, the NWP national office developed initial program features based on their 
experiences with prior programs and refined these features with their local Writing Project site 
affiliates. As part of the 2012 grant, a research team from the Education Division of SRI 
International (SRI) conducted a two-year, random-assignment evaluation of the program in 44 
rural districts across 10 states (serving approximately 25,000 students in grades 7-10), 
delivered by 12 Writing Project sites. Our evaluation found that C3WP was implemented with a 
high degree of fidelity and had a positive, statistically significant effect on students’ argument 
writing. C3WP students demonstrated greater proficiency in the quality of reasoning and use of 
evidence in their writing (Gallagher et al., 2017). A smaller study of a 1-year version of the 
program focused on rural and urban grade 7-8 teachers also found positive impacts on 
students’ argument writing (Arshan et al., 2018). 


The NWP received a 2016 i3 Scale-up grant to expand C3WP to new rural districts and grades 
4—5. Our research team designed a study to evaluate the implementation and outcomes of 
C3WP. The expanded study sought to assess the impact of C3WP on student argument writing 
after 1 and 2 years of program implementation in grades 7—10 (in 2018-19 and 2019-20) and 1 
year of implementation in grades 4—5 (in 2019-20). 


The i3 grants required evaluators to preregister design documents. These design documents 
required evaluators to lay out the design, data, and analysis plans for (1) confirmatory (i.e., 
primary) impact questions, (2) implementation fidelity, and (3) scaling indicators. The terms of 
the awards required grantees to produce and publish a public and permanent final report 
containing the results of these preregistered analyses through the Department of Education’s 
Education Resources Information Center database (for example). This technical report is 
designed to meet this final reporting requirement. 


This report reflects a substantial change from our original design. The COVID-19 pandemic 
interrupted implementation of the program and prevented the collection of student outcomes in 
spring 2020. As a result, we provide impact estimates for the effects of only 1 year of C3WP 
implementation on grade 7-9 students.* While program delivery was disrupted during spring 


1 The program was initially called the College-Ready Writers Program. NWP renamed the program to the College, 
Community, and Career Writers Program to better reflect the goals of the program. 

2 Some high schools in our study population were designed to teach ELA in semester schedules (i.e., students take 
ELA in either fall or spring). We therefore collected outcome data from approximately half of the grade 9 students in 
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2020, this report presents results of program implementation fidelity and scaling indicators 
relative to NWP goals for the grant without consideration of disruptions from COVID-19 and 
related school closures. Specifically, the report answers the following research questions. 


Impact 


e What is the effect of 1 year of exposure to C3WP on grade 7-9 students’ writing, compared 
to the business-as-usual condition? 


Implementation and scaling 
e Did C3WP meet targets for implementation fidelity? 
e Did C3WP meet targets for scaling? 


This technical report begins with a description of the NWP, C3WP’s program design and 
components, and C3WP’s intended outcomes. Then, it describes the research design, study 
sample, data, and methods. Finally, this report provides findings related to program scaling, 
implementation, and student learning. 


C3WP History and Program Design 


NWP is a network of about 175 regionally based local Writing Projects sites located at institutes 
of higher education across the U.S. and territories that support teacher PD and leadership with 
the goal of improving writing instruction. A small national office supports local sites by designing 
programs and developing resources for local sites and facilitating communications and 
relationship-building among sites. Local Writing Project sites vary widely in size, structure, and 
the types of projects they choose to engage in. 


The Common Core State Standards (CCSS) and other college-and career-ready standards, 
rolled out beginning in 2010, placed greater emphasis on writing argumentative and 
informational pieces using source materials relative to existing standards. Initially, many states 
lacked the training and PD to help teachers shift their instruction to achieve these outcomes. 
Within this context, the NWP national office received a 2012 Investing in Innovation (i3) 
Validation grant from the U.S. Department of Education to support these instructional shifts. At 
the time, NWP designed the program, known as the College-Ready Writers Program (CRWP) at 
the time, as a 2-year program to support both argument and informational writing in grades 7— 
10. The program was initially based on widely agreed-upon characteristics of high-quality PD 
without an explicit connection to instructional resources or formative assessment. Local Writing 
Project sites were asked to provide PD intensive (90 hours over 2 years); involve collective 
participation (at least 80% of English Language Arts teachers); focus on specific content (at 
least 18 hours on argument and informational writing); and use strategies designed to support 
classroom implementation (at least 8 hours of demonstration lessons, coaching, designing tasks 
and assignments, and analyzing student work). Within those broad directives, local Writing 


11 of the 47 study districts. As this rate of attrition represents a substantial degradation of the original study design, 
we omitted those findings from this report. 
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Project sites had substantial flexibility in designing programs based on their own expertise and 
their understanding of their district partners’ needs. 


Over the course of Year 1, NWP carefully monitored CRWP program implementation through 
site visits and regular check-in calls, examined early indicators of progress, and requested 
formative feedback from their external evaluators. Through this process, NWP recognized that 
teachers would need additional and focused support to ensure meaningful transfer to the 
classroom within the 2 years allotted for the program. 


For Year 2 of CRWP, NWP decided to concentrate exclusively on argument writing and to 
design instructional resources and a formative assessment tool to support teachers’ transfer of 
ideas from PD to their classrooms. They asked local sites to focus at least half of their Year 2 
PD on supporting teachers’ enactment of these materials and tools. The resulting second year 
was similar in design to the C3WP program under current study. It centered around three core 
components: 1) a set of instructional resources designed to develop students’ skills in argument 
writing; 2) ongoing, teacher-led, just-in-time PD focused on shifting classroom practice by using 
the instructional resources; and 3) regular use of formative assessment tools designed to 
assess students’ argument-writing skills and guide teachers’ next instructional steps. Local 
Writing Project sites retained responsibility for implementing the training and support for their 
regional district partners with ongoing and focused support from the national office. Our 
evaluation of CRWP, a district-randomized control trial of 44 rural districts in 10 states, 
ultimately found positive and statistically significant effects on both teacher practice and student 
source-based argument-writing achievement (Gallagher et al., 2017). In particular, CRWP 
students demonstrated greater proficiency in the quality of reasoning and use of evidence in 
their writing. 


After the initial evaluation showed positive results on student writing, NWP won a 2015 
Supporting Effective Educator Development (SEED) grant from the U.S. Department of 
Education to expand the newly rebranded C3WP’s teacher leader PD model to 90 local Writing 
Project sites across the country. The SEED grant funded a study of a 1-year version of the 
program focused on rural and urban middle school teachers, which also found positive impacts 
on students’ argument writing (Arshan et al., 2018). 


In 2016, NWP was awarded an i3 Scale-up grant aimed at testing the replication and expansion 
of C3WP. Through this grant, NWP replicated C3WP in secondary grades in new rural districts 
and extended the program to teachers in grades 4 and 5. They also executed strategies 
designed to scale the program. The logic model (Exhibit 1) lays out the key programmatic 
elements and intended outcomes of C3WP, which are described in detail below. 
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Exhibit 1. College, Career, and Community Writers Program Logic Model 


NWP Technical Assistance: 


Context for Teaching and Learning 
Policy, Curriculum, Teacher Turnover, Resources, District Initiatives 


National meetings with local site facilitators that emphasize the principles of the program and its resources. 
National Leadership Team with thinking partners assigned to each site. 
Online learning experiences for sites and teacher-consultants. 

Refinement of existing instructional and formative assessment resources; development and roll out of new upper-elementary resources. 
Ongoing support for teacher-consultants during implementation. 


Local Site Capacity Building 


Advanced Institutes that prepare local 
teacher leaders to provide C3WP ina 
high-need school through professional 
development that is and/or includes: 


Intensive and sustained, lasting 30 
hours 


Content focused on examining program 
principles, instructional resources, and 
tools: 


e Dive into instructional resources 
through model lessons, support teaching 
for at least 4 cycles of instruction in their 
own classrooms 
Bring student work into the institute for 
collaborative use of formative 
assessment 


Delivered by modeling enactment 
strategies: 
e Demonstration lessons, coaching, 


designing learning tasks, planning, and 
examination of student work. 


Program 


Professional development for teachers 
that is and/or has: 


Intensive and sustained in duration and 
breadth 


e 80% of ELA teachers participate annually 
e 45 hours of professional development 
annually 


Content focused on: 


e Argument 

e Implementing C3WP instructional 
resources that have a clear focus ona 
limited set of skills that build over time. 

e Formative assessment tools that help 
teachers determine what their students 
can do and where to focus next 
instructional steps (e.g., Using Sources 
Tool) [initial caps] 

Strategies focused on enactment: 


e Demonstration lessons, coaching, 
designing learning tasks, planning, and 
examination of student work. 
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Teacher Practice 


Actively seek out and use professional 
development resources from the local 
Writing Project site as part of their ongoing 
learning. 


Implement at least 4 cycles of 
instruction using C3WP instructional 
resources that teach students to 


e Engage in regular, and often informal, 
practices that build capacity, stamina, and 
skills for writing arguments and engaging 
in civic discourse 

e Join a conversation around a single topic 

e Use source material purposefully 

e Advance arguments with evidence 

e Apply argument skills 

e Research self-selected topics 

e Revise for an authentic purpose and 
audience 


Use C3WP formative assessment tools 
to analyze student writing to inform next 
instructional steps as a central and routine 
part of their practice 


Student Outcomes 


Performance on an on- 
demand source-based 
argument-writing task 


Content: Writing 
presents an argument 
supported by reasoning 
and developed through 
the use of evidence from 
sources 


Structure: Writing 
establishes an order and 
arrangement to enhance 
the central argument 


Stance: Writing 
establishes credibility; 
tone and style are 
appropriate for purpose 


Conventions: Writing 
demonstrates age- 
appropriate control of 
conventions 


To describe the logic model in detail, we begin with an overview of the technical assistance 
provided to local Writing Project sites to implement this program with partner districts in their 
local service areas. We next describe the local Writing Project sites’ work, which included the 
development of local “teacher leaders” and the implementation of the program in treatment 
districts with the support of these teacher leaders. We follow with a description of the teacher 
practices and student outcomes C3WP aims to influence, and finish by describing the how the 
state and district context of teaching and learning mediates the impact of the program. 


National Writing Project Technical Assistance. Provided by a small national leadership team, 
the National Writing Project technical assistance was designed to build the local Writing Project 
site leaders’ capacity to understand and implement C3WP so they could train and support their 
treatment districts in implementing the program. The C3WP national leadership team (hereafter 
known as “the national leadership team”) comprised a subset of staff from the NWP national 
office and selected leadership staff from local Writing Project sites. The national leadership 
team was responsible for overseeing the grant administration; developing and refining program 
materials; monitoring the health and progress of the C3WP work at individual sites; and 
facilitating communications and relationship-building between sites to support innovation, cross- 
pollination, and resource-sharing. 


Specific supports offered by the national leadership team included bi-annual national site and 
district-partner meetings with online learning experiences in between, the development and roll- 
out of new upper elementary instructional resources, and the refinement of existing secondary 
instructional and formative assessment resources. Members of the national leadership team 
also served as “thinking partners” to local Writing Project site leaders to provide ongoing support 
during implementation. These thinking partners provided feedback that allowed NWP to 
understand variation in implementation across sites. This role allowed the thinking partners to 
support local Writing Project site leaders who were struggling to achieve a faithful 
implementation and to disseminate information about successful innovations across sites. 


Local Site Capacity Building. Once local Writing Project site leaders received introductory 
training and resources from the national leadership team, they became responsible for 
identifying and training local teacher leaders who could train partner districts to implement 
C3WP Teacher leaders were predominately current or recent teachers who had previous 
involvement with local Writing Project sites. NWP’s theory of action posits that teacher leaders 
need to deeply understand the program principles and instructional resources so they can 
effectively help other teachers implement the approach in the classroom. Further, the program 
posits that the most effective way to understand the program principles and resources is direct 
experience teaching C3WP to students, with the opportunity to plan and reflect on the 
experiences collaboratively. Each local Writing Project site hosted two intensive, ongoing PD 
experiences called Advanced Institutes one the year before starting work with grades 7—10 
teachers and another the year before starting work with grades 4—5 teachers. The Advanced 
Institute included the following features: 
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e 30 hours of intensive and sustained PD that typically included a summer session held over 
1-2 weeks and sessions throughout the school year to support teacher leaders while they 
implemented four cycles of C3WP instruction (defined below) in the classroom. 


e Content covering C3WP program principles, resources, and PD strategies aimed at 
supporting classroom enactment. 


e Collaborative use of C3WP formative assessment tools to examine student work following 
each cycle of instruction. 


The national leadership team expected that each local Writing Project site would continue 
building its pipeline of teacher leaders to deliver the program as dictated by their local needs 
and continue running local Advanced Institutes as new teacher leaders were recruited. 


C3WP Components.* C3WP has three, key interwoven program components to support 
classroom implementation: intensive PD, instructional resources, and formative assessment. 
When implemented in a cyclical sequence, these three elements are known as a cycle of 
instruction. A C3WP cycle of instruction is an iterative process that involves several steps: 1) 
teachers engage in PD to learn how to implement an instructional resource; 2) teachers 
implement the instructional resource in their own classroom, resulting in student writing; 3) 
teachers use the students’ writing to assess student progress; and 4) teachers select the next 
instructional resource based on students’ needs. Step 4 starts the next cycle of instruction. 


e Intensive professional development to support classroom implementation. To 
effectively support implementation, C3WP PD was designed to be sustained, job- 
embedded, and focused on classroom enactment; that is, evidence-based PD strategies 
that support shifts in teacher practice (Darling-Hammond et al., 2017). These evidence- 
based strategies PD strategies included demonstration lessons, co-teaching, and co- 
planning. Teacher leaders supporting the treatment teachers operationalized a core NWP 
principle: teachers who are well informed and effective in their practices can successfully 
teach other teachers (Gray, 2000; Lieberman & Friedrich, 2010; Lieberman & Wood, 
2003). The C3WP approach called for PD to be on-going and intensive in duration. 


e Content focused on the use of instructional resources and argument. C3WP’s 
instructional resources are designed to guide classroom implementation. With these 
resources, the national leadership team aimed to operationalize abstract ideas and transfer 
new content knowledge into teacher instruction, while the PD provided critical scaffolds for 
rapid implementation of complex and often new approaches to instruction. The core of the 
instructional resources are multi-day units focused on developing specific argument skills. 
These multi-day units are designed to build on each other, beginning with writing a claim 
and building to more advanced skills that may depend on the earlier units, such as revising 
claims or ranking evidence. These multi-day, skill-based resources include text sets: 


3 NWP provides program description and links to all instructional resources at 
https://sites.google.com/nwp.org/c3wp/home 
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approximately four nonfiction sources representing multiple points of view on a single topic. 
While these multi-day, skill-based resources serve as the core of C3WP instruction, the 
program also provides three other types of resources to supplement, reinforce, and extend 
the program’s reach. These resources include: 


- Supplemental scaffolds, including short lessons used to revisit or reinforce a particular 
skill, or individual resources such as graphic organizers. 


- Routine Argument Writing, which are regular, skill-based writing exercises that can be 
implemented between cycles of instruction. They are designed to take 5—7 minutes and 
build students’ understanding of argument and stamina with argument writing. 


- More advanced multi-day resources on independently chosen topics of interest to 
support students’ bringing together multiple argument skills (e.g., writing an op-ed on an 
issue of local importance). 


e Formative assessment to inform instruction. A key feature of C3WP’s design was 
regular analysis of student work to show teachers how students were progressing and help 
teachers plan for future lessons. NWP designed two C3WP formative assessment tools to 
support teachers with analyzing student work: the Using Sources Tool and the Claim, 
Evidence, and Reasoning Protocol. These tools generated rapid-cycle feedback on student 
learning and focused teachers on aspects of writing specific to the skills and approaches 
emphasized in C3WP resources. The Using Sources Tool, by far the more commonly used 
of the two tools, asks teachers to rate and describe the writing’s claim (e.g., if the claim is 
nuanced and/or debatable) and the use of source material (e.g., describing how students 
connected the source material to their claim (Harris, 2006). 


These three components are designed to reinforce each other by supporting the development of 
teachers’ content knowledge and scaffolding the development of new instructional practices. As 
implemented, these components typically work together; local Writing Project sites often provide 
in-classroom supports for teachers implementing the instructional resources (e.g., co-teaching, 
coaching). Teachers typically bring resulting student work to PD sessions, where formative 
assessment takes place collaboratively as the local Writing Project site leaders help teachers 
understand the data and plan the next steps for instruction. 


Variation is central to NWP’s program model. While committing to the three key program 
components described above, local Writing Project site leaders had flexibility in implementing 
C3WP based on their own expertise and sense of their district partners’ needs. Thus, local 
Writing Project sites conducted annual needs assessments and solicited partner districts’ input 
on their PD plans. By promoting adaptation of the core model, NWP set out to tailor the program 
to the local context and foster commitment to reform. 


Teacher Practices. The program calls on teachers to implement at least four C3WP cycles of 
instruction to build students’ argument-writing skills, with the assumption that each cycle will 
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incorporate one of the multi-day units. Throughout the year of C3WP implementation, teachers 
are asked to engage students in Routine Argument Writing. 


Student Learning Outcomes. C3WP aims to improve students’ ability to write arguments that 
effectively use evidence from nonfiction source material to support a claim. The program breaks 
the argument-writing process into discrete skills, such as identifying and responding to 
arguments, organizing evidence, selecting and annotating evidence, and crafting a claim with 
evidence. The expectation is that students who learn these skills will perform better on source- 
based argument-writing tasks. C3WP also seeks to help students develop new habits of mind, 
including “reading critically, exploring multiple points of view, and taking a stand on important 
issues” (Friedrich et al., 2018, p.19). 


Context for Teaching and Learning. The C3WP program was implemented in diverse districts 
across the country with varying contexts for teaching and learning. As represented across the 
top of the logic model (Exhibit 1), various contextual factors are expected to influence the work. 
Important factors at the policy level include the state standards, presence of writing and 
argument-writing on state and local assessments, school accountability, and teacher-evaluation 
systems. At the district level, curriculum, teacher turnover, availability or lack of resources, and 
framing around C3WP PD in relation to competing priorities shapes teachers’ attitudes and 
practices. 


Scaling 


NWP’s approach to scaling aimed to build a national infrastructure to enable the spread of the 
program to new schools and districts while maintaining the integrity of the program. NWP had 
four main strategies for scaling the program: 1) expanding NWP’s national leadership capacity; 
2) developing and refining instructional and formative resources; 3) investing in regional 
leadership; and 4) developing site certification and micro-credentialing systems. For each of 
these strategies NWP set goals as indicators to ensure they reached those goals. We describe 
each of the strategies in more detail below. 


Expand NWP National Leadership Capacity. As described above, the national leadership 
team designed C3WP technical assistance to support the development of local capacity, with 
the expectation that sites could implement C3WP PD independently at the grant’s conclusion. In 
addition to this site-capacity building, NWP looked to ensure program strength and continuity 
through the development of a leadership pipeline by providing local Writing Project site leaders 
opportunities to lead cross-site development (e.g., lead a C3WP training at an NWP national 
meeting, serving as a thinking partner for a new site, or collaborating on resource development). 
Through capacity-building efforts, NWP aimed to develop a cadre of new leaders experienced in 
leading C3WP work at the national level. Part of the capacity-building approach was to establish 
a marketing strategy and professionally designed website to support the dissemination of 
C3WP. 
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Develop/Refine Instructional and Formative Assessment Resources. The national 
leadership team developed and refined C3WP design principles and instructional and formative 
assessment resources. The following instructional resource design principles operationalize 
C3WP’s approach to argument-writing instruction: 


e Focus ona specific set of skills or practices in argument writing that build over the course of 
an academic year; 


e Provide text sets that represent multiple perspectives on a single topic, beyond pro and con; 


e Engage students in iterative reading and writing practices that build knowledge about a 
conversation focused on a single topic; 


e Support the writing of recursive claims that emerge and evolve through the reading and 
writing process; 


e Support students in creating intentional organizational structures that are designed to 
advance the argument, not based on formulas (e.g., five-paragraph theme); and, 


e Provide formative assessment opportunities embedded in classroom practice and 
professional conversations that identify areas of strength and inform the next steps for 
teaching and learning. 


This process was designed to ensure that high-quality materials were available to support the 
teaching of argument writing and ensure that adaptations to program materials would have clear 
guidelines to support program fidelity. 


The national leadership team also aimed to develop a set of resources for grades 4—5 that 
taught a progressive set of argument-writing skills similar to the set for grades 7-10. In addition, 
they refined the resources for grades 7-10 so they reflected the design principles. 


Invest in Regional Leadership. NWP invested in the development of regional leadership by 
funding local Writing Project sites to hold Advanced Institutes through which they trained 
teacher leaders to implement and provide PD for C3WP. NWP also funded local Writing Project 
sites to offer C3WP PD in partner districts, which served to scale up and increase the number of 
districts involved and build the capacity of local Writing Project sites to facilitate PD. The local 
Writing Project sites received ongoing technical assistance and support from NWP and learned 
through practice. 


Develop Site Certification and Micro-Credentialing System. To cultivate the depth of change 
needed to support and sustain local C3WP implementation, NWP developed a certification and 
micro-credentialing process to identify sites qualified to offer C3WP and credential s. NWP 
developed criteria and processes for certifying sites to offer C3WP and to award micro- 
credentials to teachers. The three-level micro-credentialing system designated a teacher 
leader's required level of experience to provide C3WP PD. By investing in regional leadership 
development and the new certification and micro-credentialing system, NWP hoped to certify 
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local Writing Project sites to offer C3WP, identify model sites, and provide teacher leaders with 
a micro-credential in C3WP at one of the three levels. 


Exhibit 2. C3WP Scaling Strategy and Indicators 


Strategies for Developing a 


National Infrastructure 


Expand NWP National Leadership Capacity 
Develop capacity of national support team 
Hold virtual meetings and in-person 
convenings 
Develop marketing strategy and professionally 
designed, public facing website to support the 
broad usage of C3WP Open Educational 
Resources 


Develop and Refining Instructional and 
Formative Assessment Resources 

¢ Make explicit design principles 

* Revise existing instructional resources 

* Expand resources to upper elementary grades 


Invest in Regional Leadership 
* Fund intensive Advanced Leadership 
Development through Advanced Institutes 
— Implement instructional resources 
— Engage in collaborative use of formative 
assessment tools 
— Prepare teachers to lead year-long 
professional development through 
examination of program principles, 
resources, and models of coaching, co- 
teaching, and co-planning 
¢ Fund local Writing Projects to facilitate C3WP 
in high-needs districts 


Develop site certification and micro- 

credentialing system 

¢ Develop criteria and processes for certifying 
sites to offer C3WP 

* Develop criteria and process to allow sites to 
micro-credential teachers in C3WP 


Network Indicators of 
Patel a=t-kx=Xe meter (= 


Expansion Goals By 2021: 


Increased NWP National Capacity to Support 

C3WP 

* New teacher leaders with experience 
providing national leadership (e.g., leading a 
training at an NWP national meeting): 12 


Dissemination of C3WP Resources 
¢ Users of C3WP Open Educational Resources: 
50,000 


Increased Quantity of Quality Instructional 

Resources 

* Resources for grades 7-10 standardized in 
format and revised with an eye for 
accessibility 
New full set of resources for grades 4-5 
representing the full arc of the program 


Increased Number of Certified Sites Providing 

C3WP PD 

* Sites certified to offer C3WP (through 
additional grant funding or through fee-for- 
service, etc.): 16 

* Identification of C3WP model sites: 3 


Increased Number of Credentialed Teacher 

Leaders Skilled in Delivering C3WP 

* Teacher leaders who complete 30+ hours of 
Advanced Institute: 200 


Research Design 


Overview 


The study employed a district-level cluster, randomized, controlled trial to assess the impact of 
C3WP on students’ source-based argument-writing achievement. As initially designed, the study 
intended to measure the impact of 2 years of C3WP on grade 7-10 student outcomes (in the 


Evaluation of the College, Career & Community Writers Program | February 2021 12 


2018-19 and 2019-20 school years), and 1 year of C3WP on grade 4—5 student outcomes (in 
the 2019-20 school year). In March 2020, the COVID-19 pandemic interrupted the study’s data 
collection plans. Student outcome data could not be fully collected in spring 2020 from 
secondary students completing their second program year or from elementary (4—5) students 
completing their first program year.* This report focuses only on implementation fidelity over the 
full study period, the one-year impacts of the program on students in grades 7-9, and indicators 
of scale.® 


Recruitment, Randomization, and the Counterfactual Condition 


NWP invited 17 local Writing Project sites to participate in this evaluation. Each site recruited 
one or more pairs of districts in their service area as partners for the study. To be eligible to 
participate, districts had to have a majority of rural schools according to the guidelines issued by 
the i3 grant funding the project (i.e., having a federal locale code 32, 33, 41, 42, or 43). Local 
Writing Project sites were directed to recruit pairs of districts of similar size, prior achievement, 
and demographics, with the understanding that one district in each pair would be assigned to 
treatment, the other to control. 


Such blocked randomization has several benefits. First, local Writing Project sites can control 
how many districts they serve in a year. Second, blocked randomization provides better balance 
across treatment conditions (accounting for prior achievement, local policy, etc.) than 
randomization without regard to blocks. 


Before random assignment, districts committed to participating in data collection and remaining 
in the randomly assigned study condition. Specifically, district leaders committed to supporting 
teacher participation in C3WP PD and use of C3WP materials in the classroom if randomized 
into the treatment condition. If randomized into the business-as-usual group, districts also 
committed: 1) to continue typical instructional practice and PD programming (as if they had not 
heard of C3WP), 2) not to seek out similar PD (i.e., with a focus on argument writing, 3) to wait 
until after the study to use any C3WP tools, materials, or strategies. Control districts received an 
incentive of $2500.00 each year of the study and offered C3WP the year following the 
experiment during the 2020-21 school year. 


In spring 2018, the research team randomized one of each recruited pair of districts into 
treatment and assigned the other to the control condition. Districts randomly assigned to 
treatment received C3WP training in 2018-19 and 2019-20. Secondary (grade 7-10) teachers 
in the treatment districts received 2 years of C3WP support, beginning in summer of 2018. 
Upper elementary (grade 4—5) teachers in the treatment districts received 1 year of C3WP 
support, beginning in summer of 2019. 


4 A small number of secondary prompts were collected students who took ELA during the first semester. 
5 The full evaluation included a number of additional components, including teacher instructional logs and surveys to 
understand treatment-control contrast and impacts on teacher practice, and qualitative data collection to understand 
program scaling. These data sources, research methods, and resulting findings are reported elsewhere. 
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The study as designed, executed, and described in this technical report closely mimics the 
design pre-registered as part of the National Evaluation of i3 with one notable exception: we 
designated student outcomes from spring 2020 (impacts of 2 years of C3WP on secondary 
students and 1 year of C3WP on elementary students) as our two confirmatory impacts. As 
these data were not collected due to COVID-19-related school closures, we report on students 
impacts after 1 year of C3WP on secondary students. While our pre-registered design is not 
publicly available, the project proposal is publicly available and may serve a similar purpose in 
providing transparency in research design.® 


Study Samples (WP, District, Schools, Teachers, Students) 


Writing Project Site and District Samples 


NWP recruited 17 local Writing Project sites with histories of providing in-service development to 
schools and districts. The local Writing Project sites spanned 16 states: Alabama, California, 
Idaho, Kentucky, Michigan, Mississippi, Missouri, Nebraska, Ohio, Oklahoma, South Carolina, 
Tennessee, Texas, Virginia, Washington, and Wisconsin. To be eligible to participate, all 
districts had to meet the federal definition of rural districts. The randomized sample comprised 
48 school districts. According to the 2016—17 Common Core of Data (CCD) from the U.S. 
Department of Education, study district enrollment ranged from 147 students to 5,817 students. 
The majority of students at 45 of the 48 were eligible for free or reduced-price lunch.’ The 
populations in seven of the study districts were majority African American, one was majority 
American Indian/Alaskan native, six were majority Hispanic, and thirty were majority White. At 
the remaining four districts, no single racial or ethnic group comprised a majority of the student 
population. Exhibit 3 shows the student demographics averaged at the district-level. Exhibit 4 
shows the student demographics averaged at the student-level. One district attritted in spring 
2018, after being randomized into the treatment condition. The attrition occurred before either 
program implementation or baseline data collection began. 


Exhibit 3. District Demographics in 2016—17, District-level Averages 


Treatment Control Overall 
Total Enrollment 1628 1789 1709 
% Free or Reduced-Price Lunch 68.1 71.1 69.6 
% English Language Learners 8.7 49 6.8 
% American Indian/Alaska Native 4.9 1.9 3.4 
% Asian 0.4 0.5 0.4 
% Black, Non-Hispanic 16.9 14.7 15.8 
% Hawaiian Native/Pacific Islander 0.1 0.1 0.1 
% Hispanic 13.9 16.6 15.3 


6 NWP and SRI’s proposal is available by request from the authors or at: 
https:/Awww2.ed.gov/programs/innovation/awards. html 

7 As of 2016-17, between 32.7% and 95.5% of students in sample districts had students who were eligible for free 
and reduced-price lunch (CCD 2016-17). 
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Treatment (Oey aii Ke) | Overall 
% Two or More Races 4.3 2.6 3.5 
% White, Non-Hispanic 59.4 63.7 61.6 


Note. This table counts each district as one and averages them. 


Exhibit 4. District Demographics in 2016—17, Student-level Averages 


Treatment lofey alice) Overall 

Total Enrollment 1628 1789 1709 
% Free or Reduced-Price Lunch 76.4 31 75.0 
% English Language Learners 9.1 3.6 6.2 

% American Indian/Alaska Native 2.1 0.6 1.3 

% Asian 0.5 0.4 0.4 

% Black, Non-Hispanic 26.1 22.5 24.2 

% Hawaiian Native/Pacific Islander 0.1 0.1 0.1 

% Hispanic 7.2 10.8 9.1 

% Two or More Races 3.3 2.2 Zl 

% White, Non-Hispanic 60.7 63.4 62.1 


Note. This table counts each student. 


Teacher Samples 


To be eligible for inclusion in the study, teachers had to be a core ELA teacher in grades 4—5 or 
7-10 in one of the study districts. For secondary grades, a core ELA teacher was a teacher who 
taught an ELA class used to meet graduation requirements. Excluded teachers were those who 
taught only elective classes, such as journalism, self-contained special education classes, and 
classes designed for English learners who are not tested in English on the state assessment. 


For upper elementary grades, used for the implementation study only, we included the primary 
classroom teachers or the teachers responsible for ELA if the grade was departmentalized. 


Across the entire teacher sample, we excluded co-teachers, paraprofessionals, and associate 
teachers. 


Student Samples 


Study districts completed the baseline student writing assessment within the first five weeks of 
the 2018-19 school year. These baseline student assessments determined student eligibility for 
the study-assigned sample. No students joining a district after the baseline student writing 
assessments were collected included in the assigned or analytic sample. Both the assigned and 
analytic student samples only include students from non-attrited schools. 


To provide an unbiased sample and provide an estimate of student attrition, we randomly 
sampled students from the pretest student population into an assigned sample whether or not 
they completed post-test writing. The analytic sample, therefore, provided an unbiased sample 
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of students who remained in the district (and completed both the pretests and posttests) 
between the beginning and end of the 1-year implementation. 


To achieve the desired analytic sample size in the summer of 2020 (after the posttest), we 
added to the assigned sample until we had 15 students per district, per grade in the analytic 
sample (i.e., with both pretest and posttest student writing). Students in the assigned sample 
without posttest writing were considered attritions. 


Nassi ned_sample ~ ‘Yanalytic_sample 
Attrition = GSS G Ned SOM ple GnGtEle SOD re. 
Nassigned_sample 


Student attrition was 48.8% overall, 49.9% in the treatment group and 47.6% in the control 
group (Exhibit 5). This combination of overall and differential attrition meets the What Works 
Clearinghouse’s optimistic boundary for attrition (What Works Clearinghouse, 2020), indicating 
that bias from attrition is unlikely to degrade the quality of the study’s internal validity. 


Exhibit 5. Impact Study Attrition, Grade 7-9 Impacts of 1 Year of C3WP 


PACT J(efar-ve| Analytic Assigned Analytic 
District District District Student Student Student 
Sample Sample Attrition Sample Sample Attrition 
Treatment 24 23 4.2% 1,861 932 49.9% 
Control 24 24 0.0% 1,763 924 47.6% 
Overall 48 47 2.1% 3,624 1,856 48.8% 
Differential 2.1% 2.3% 


NWP was unable to score 113 baseline papers (due to, e.g., papers being completed or 
scanned in a way that scorers could not read or being misplaced during storage or 
transportation). We imputed missing baseline data for these records using a constant (the 
sample mean) and an indicator equal to 1 for students whose baseline score was imputed. Such 
an approach is unlikely to introduce bias in a randomized control trial design when used for 
baseline data (What Works Clearinghouse, 2020). 


Data and Methods 


We used various data sources, including administrative program data, teacher surveys, and an 
on-demand writing assessment, to examine scaling, program implementation, and student 
outcomes.® The following information describes these measures in additional detail. 


Program Implementation Fidelity 


We assessed two aspects of program implementation as part of our evaluation: development of 
the local Writing Project sites’ capacity to deliver C3WP PD in the 2017-18 and 2018-19 school 


8 Teacher instructional practice measures were also collected through teacher instructional logs, but teacher 
outcomes were not a focus of this report. 
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years and the implementation of the program in treatment districts in the 2018-19 and 2019-20 
school years. The implementation fidelity measures and thresholds reflect the metrics found in 
sufficient dosage to impact student outcomes (Gallagher et al., 2017). For each of the four 
components, thresholds were set at the teacher, district, and/or site level and rolled up to 
assess implementation at the overall program level. The first component, which measures 
capacity building, was measured in the 2017-18 and 2018-19 school years. The latter three 
components, which measure implementation of the program in treatment districts, was 
measured in the 2018-19 and 2019-20 school years. Each component is described below. 


Local Writing Project Capacity Building 


Sites developed local capacity by holding Advanced Institutes to prepare teacher leaders to 
provide PD in C3WP in the year prior to starting work with the treatment district. To meet 
implementation fidelity for the local capacity building component, sites had to meet thresholds 
for three elements in the 2017-18 and 2018-19 school years: (1) Duration and breadth of 
participation, (2) use of formative assessment tools, and (3) exposure to C3WP instructional 
resources. Exhibit 6 itemizes the implementation fidelity measures and thresholds at the site 
level. To meet program level implementation fidelity, 75% of sites reach each site-level 
threshold. 


Exhibit 6. Component 1: Implementation Fidelity Local Capacity Building Metrics 


Operational Definition 


Key Element of the Indicator Site-Level Threshold Years Measured 
Duration and breadth Soa nears esieniay 5 teacher leaders met 
of participation a Al P threshold 


; : Site submits student 
Use of formative eile SUD ini St dent work work analysis via 2017-18 and 2018-19 


analysis via NWP’s UST ; 
assessment tools Ana Oceasione In Al NWP's UST on2 school years 
occasions in Al 


Exposure to C3WP At least 5 teacher leaders 


‘aetuctional experience 4 C3WP 4 C3WP instructional 
instructional resources in resources covered in Al 
resources Al 


Note. Al stands for “Advanced Insitute” and UST stands for “Using Sources Tool.” Measured across two Al’s, across 
the 2017-18 school year and the 2018-19 school year. Target teacher leaders were teachers who do not work in the 
study districts. This capacity buiding was intended to prepare teacher leaders to support C3WP PD delivery in 
treatment districts in the 2018-19 and 2019-20 school years. 


C3WP District Implementation 


Implementation fidelity metrics also assessed the extent to which the C3WP PD local Writing 
Project sites delivery of C3WP PD in treatment districts in 2018-19 and 2019-20 aligned with 
the key program’s key components: intensive PD to support classroom implementation, content 
focused on argument, C3WP instructional resources, and formative assessment. 
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Duration and breadth of PD. During the first year of PD, sites were expected to provide 45 hours 
of PD to treatment districts’ ELA teachers in grades 7-10. In the second year, sites were 
expected to provide an additional 45 hours to treatment districts’ ELA teachers in grades 7-10 
and 45 hours to treatment districts’ teachers responsible for ELA in grades 4—5. Exhibit 7 lays 
out the implementation fidelity measures and thresholds for the 2018-19 and 2019-20 school 
years. 


Content of PD. C3WP PD was expected to focus primarily on argument writing, include use of 
the C3WP instructional resources, and include formative assessment of student work. This 
component included three elements: 


e Focus on argument writing. The expectation was for 80% of teachers to participate in 36 
hours of argument-writing PD annually. 


e Use of C3WP instructional resources. Sites were expected to cover at least four of the multi- 
day C3WP resources in PD annually. 


e Use of formative assessment tool. Sites were expected to use the Using Sources Tool in PD 
with teachers at least twice each year. 


Exhibit 8 lays out the implementation fidelity measures and thresholds for the content of PD 
delivered to treatment districts’ teachers in the 2018-19 and 2019-20 school years. 


Enactment-focused PD strategies. C3WP’s approach to PD focused on supporting teachers to 
shift their classroom practice by using the instructional resources and formative assessment 
tool. Fifty percent of PD events were expected to use enactment-focused strategies, such as in- 
class demonstrations with students or planning for classroom implementation. Exhibit 9 lays out 
the implementation fidelity measures and thresholds for the strategies used to deliver C3WP PD 
to treatment districts’ teachers at the teacher- and district-level thresholds. 


To meet the program-level threshold for each component, at least 75% of districts had to meet 
the district-level thresholds. 


Exhibit 7. Component 2: Duration and Breadth of Professional Development Implementation 
Fidelity Metrics 


(oyey-Ve-litoyar-1mPL-vilaliacey a} Teacher-Level District-Level Years 


Key Element of the Indicator Threshold Threshold Measured 


85%+ of district's 
target teachers 
reach teacher- 


level threshold A 
Duration and 80% of target teachers Ae 35% hours OR 2018-19 and 


breadth of participate in 45 or more 2019-20 


participation hours of PD annually B: 40+ hours 75% of district's school years 
target teachers 


reach teacher- 
level threshold B 
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Key Element 


foyer-Ve-litoyar-1MmPL-vilaliace a} 
of the Indicator 


Teacher-Level 
Threshold 


District-Level 
Threshold 


Years 
Measured 


Duration and 
breadth of 

participation 
over 2 years 


80% of 7-10 ELA 


teachers participate in 90 A: 70+ hours 


or more hours of PD 
over 2 years 


B: 80+ hours 


85%+ of 7-10 
ELA teachers 
reach teacher- 
level threshold A 


OR 


75% of 7-10 ELA 
teachers reach 
teacher-level 
threshold B 


2018-19 and 
2019-20 
school years 


Note. Target teachers for 2018-19 included core 7-10 ELA teachers. Target teachers for 2019-20 included core 7— 


10 ELA teachers and teachers responsible for ELA in grades 4-5. 


Exhibit 8. Component 3: Content of Professional Development Implementation Fidelity Metrics 


Key Element 


Operational Definition 


Teacher-Level 


District-Level 


Years 


oy im tat=Mlarel(ercine) el 


80% of target teachers 


Threshold 


Threshold 


85%+ of district's 
target teachers 
reach teacher- 


Measured 


2018-1 
Focus on participate in 36 hours ay. 954 hours level threshold A ee el 
argument of PD on argument- OR hool 
writing writing instruction eye 75%+ of district’s ae al 
annually target teachers 
reach teacher- 
level threshold B 
80% of target teachers 4 C3WP 2018-19 and 
epee use at least 4 C3WP N/A instructional 2019-20 
Keeoutces instructional resources resources school years 
in PD annually covered in PD 
Site submits 
Use of Analysis of student a ae 2018-19 and 
formative work with UST occurs N/A district via NWP’s 2019-20 
assessment during PD at least twice online UST on 2 school years 
tool annually 


occasions by the 
end of February 


Note. UST stands for “Using Sources Tool.” Target teachers for 2018-19 included core 7-10 ELA teachers. Target 
teachers for 2019-20 included core 7-10 ELA teachers and teachers responsible for ELA in grades 4—5. 


Year 1 Impacts of the College, Career & Community Writers Program | February 2021 


Exhibit 9. Component 4: Professional Development Strategies Implementation Fidelity Metrics 


Key Operational Definition of the Uebel Pies Years 


; Level Level 
Element Takelber-1ke) g Threshold Threshold Measured 


For 80% of target teachers, 50% of 
PD events focus on classroom 


enactment annually, including (1) in- 75%+ of 

class demonstration with students, (2) district's 2018-19 and 
Focus on designing tasks/assignments, (3) 50% of PD target 2019-20 
classroom planning for classroom events teachers school years 
enactment implementation, (4) analyzing student reach 

work, (5) modeling instruction with teacher-level 

teachers, (6) co-teaching/co-planning, threshold 

or (7) debriefing classroom 

implementation 


Note. Target teachers for 2018-19 included core 7-10 ELA teachers. Target teachers for 2019-20 included core 7— 
10 ELA teachers and teachers responsible for ELA in grades 4-5. 


Data 


Implementation fidelity data was supplied by NWP and came from two administrative data 
sources. NWP’s Professional Learning Tracker housed information on PD hours, content, and 
strategies for each local Writing Project site. Local sites submitted Professional Learning 
Tracker data three times a year to NWP’s national office. NWP created an online platform for 
C3WP’s formative assessment tool, the Using Sources Tool. Teachers were expected to enter 
data for students after each cycle of writing that would be reviewed in PD. 


Scaling 


Exhibit 10 shows the measure and minimum threshold for each of the scale-up goals in the 
C3WP scaling logic model. We assessed the extent to which NWP reached these scale-up 
goals by February 2021, though NWP continues to work on scaling activities beyond the 
timeline of this report. Data for each of the measures came from diverse sources and included 
review of C3WP documents such as meeting agenda, leadership team lists, C3WP website 
metrics and materials, and the Professional Learning Tracker data. 


Exhibit 10. College, Career, and Community Writers Program Scaling Measures and Thresholds 
by Goal 


Minimum 
Yor-1 (=U) ok CTey-1 | Measure Thteshold 
: ; Agendas/programs for national meetings 
New teacher leaders with experience OR 
providing national leadership (e.g., ae — ; err 
Invitations to join C3WP leadership team 12 individuals 


leading a training at an NWP national 


meeting) for individuals not on leadership team 


during the i3 Validation grant 
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Minimum 
sYor- | (sta 0) ok CTey-]| Measure 


OM alass-yare)(e| 
Users of C3WP Open Educational Google and website analytics for page 50,000 page 
R P views, collected annually by NWP and pe gies 
esources views 


provided to the research team 


Resources for grades 7-10 


standardized in format and revised for Number of resources posted on the 


20 resources 


accessibility Cone ene 
Full year arc of resources that shows a 4 Fulsator 
New resources for grades 4—5 progressive set of argument-writing skills 
; resources 
posted on C3WP website 
identification of C3WP model sites Local Writing Project sites that co-lead 3 local Writing 
regional C3WP conferences Project sites 


Sites certified to offer C3WP (through Local Writing Project sites holds C3WP 
additional grant funding or through fee- = Als and lead 2 years of C3WP PD with 
for-service, etc.) partner school 


16 local Writing 
Project sites 


200 teacher 
leaders 


Teacher leaders who complete 30+ 


heures Al Professional Learning Tracking Form 


Note: Al stands for Advanced Institute. 


Student Writing Assessment 


We used an on-demand writing assessment as our primary outcome. 


Data 


To measure students’ writing ability, the research team worked with study districts’ staffs to 
administer on-demand, source-based, argument-writing assessments in each study district. We 
collected prompts from the grade 7-9 students in fall 2018 (baseline) and spring 2019 (impacts 
after 1 year). Students completed the assessment over two days, in 45-minute blocks. On the 
first day, students were given a packet containing multiple short, non-fiction source texts about a 
single topic. On day two, students received these source materials back (along with any notes 
or pre-planning they had conducted) and a prompt to write an argument using evidence from the 
texts. Students received one of five topics. For example, one prompt presented six texts about 
the impact dams have on salmon migration, including an informational text introducing the 
issue, potential solutions, and varying perspectives on the costs and benefits of both the dams 
and the potential solutions. On the second day, students were tasked with writing an argument 
about whether the dams on the river should be destroyed to save migrating salmon for the 
Director of the Army Corp of Engineers, who is responsible for managing the dams. 


The student writing was returned to the research team, who de-identified the writing to protect 
student privacy and blind scorers to treatment condition and administration time. Papers were 
scored via a virtual scoring conference at the conclusion of the impact study using the Analytic 
Writing Continuum for Source-based Argument (AWC-SBA). 


The AWC-SBA was developed from the NWP’s Analytic Writing Continuum (AWC), a valid and 
reliable measure of student writing (Bang, 2013), to measure source-based argument writing. 
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AWC-SBA retains the AWC’s basic structure (rooted in the “six traits” of writing) but focuses on 
the particular attributes of source-based argument writing. The AWC-SBA measures four 
attributes: content (quality of reasoning and strength of evidence); structure (organization to 
enhance the argument); stance (tone, establishment of credibility); and conventions 
(differentiation of source material from original writing, punctuation, spelling, capitalization, and 
paragraphing). 


Exhibit 11 shows the baseline descriptive statistics by treatment status on the four AWC-SBA 
components. We did not perform missing imputation of baseline scores to calculate these 
averages, and the values reflect the analytic sample’s baseline raw AWC-SBA component 
scores. 


Exhibit 11. Average Baseline Score on the AWC-SBA, by Treatment Condition 


Treatment Control Baseline 
Baseline Overall Pooled Equivalence 
Measure Mean SD SD Mean SD ((al=te fe (-t-mre)) 
Content 2.45 1.05 868 2.44 1.05 875 2.44 1.05 0.01 
Stance 2.46 1.06 868 2.47 1.08 875 2.47 1.07 -0.01 
Structure 2.36 1.04 868 2.38 1.06 875 2.37 1.05 -0.01 
Conventions 2.50 1.13 868 2.52 1.12 875 2.51 1.13 -0.02 


To estimate reliability, we randomly selected 595 student prompts to be scored by two raters to 
provide evidence of inter-rater reliability. Exhibit 12 shows the number and percent of student 
prompts that were double scored and received scores within 1 point (N=595) by attribute. 


Exhibit 12. Interrater Reliability by AWC-SBA Attribute 


Number of papers Number of student Percent of scores 
within 1-point prompts that were within 1-point 
agreement ro [ol] 0) (-mcyere] c-Ye | agreement 
Content 520 595 87% 
Stance 513 595 86% 
Structure 513 595 86% 
Conventions 502 595 84% 
Model 


To assess the impact of C3WP on student achievement, we estimated a Hierarchical Linear 
Model (HLM) to adjust standard errors associated with the clustering of observations within 
districts (Raudenbush & Bryk, 2002). Models are estimated separately for each of the four 
attributes scored. The predicted source-based argument-writing score for student /, in district /, 
in randomization block k as a function of attending a district assigned to treatment is given as 


Yijx = Bo + Bi(C3WP;) + (Covariates, B2 + pit nit; 


Evaluation of the College, Career & Community Writers Program | February 2021 22 


Random effects n;and 4; allow for error at the student and district level, respectively. We include 
pi, a vector of randomization block fixed effects, which we entered as a series of indicator 
variables and centered within the analytic sample. We include a vector of student-level 
covariates to improve the precision of the estimate. These covariates include the student’s 
baseline score on each of the four attributes, an indicator variable for the individual who scored 
their outcome paper, and an indicator variable equal to 1 if the students’ baseline scores were 
imputed using the sample mean. We centered baseline and outcome AWC-SBA scores within 
each student’s grade and prompt form (using, in both cases, overall sample means and 
standard deviations) to account for cohort and prompt effects. We centered scorer indicators 
and the missing dummy indicator within the analytic sample. #1 provides an estimate of the 
intent-to-treat effect of C3WP on a participating student’s writing ability. 


At the study outset, we specified the content attribute at the end of program Year 2 as our 
confirmatory impact estimate for secondary grades. While we were unable to collect the Year 2 
impact data, we consider the content attribute for our Year 1 secondary grades to be our 
equivalent confirmatory estimate. We also acknowledge that, since we wrote this proposal, the 
What Works Clearinghouse released an adolescent literacy protocol intended to guide review of 
studies estimating the impacts of programs on grade 4—12 student literacy. According to this 
protocol, the AWC-SBA attributes would fall under two domains: writing quality (content, 
structure, and stance) and writing conventions (conventions). We therefore test the three 
measures of writing quality against the critical p-values determined by the Benjamini-Hochberg 
multiple comparison correction. Since there are three comparisons within the same domain, we 
therefore compare the highest of these three p-values against .017, the next highest against 
.025, and the lowest against .050. 


Findings 


This section presents findings on the extent to which C3WP was implemented as initially 
intended and describes the impact of the program on grade 7—9 student outcomes. 


Implementation 


C3WP met its implementation fidelity metrics for all components in each year, except for the 
intensity of PD in Year 2. The spring of Year 2 coincided with the start of the COVID-19 
pandemic, which impacted local Writing Project sites leaders’ ability to conduct planned PD 
activities. 


Most local Writing Project sites (76%) met the thresholds for the local capacity-building 
component, indicating that most sites had prepared a group of teacher leaders to support C3WP 
PD through their Advanced Institute. Overall, C3WP met the program-level threshold of 75% of 
sites meeting the three local capacity-building indicators (see Exhibit 13). 


Year 1 Impacts of the College, Career & Community Writers Program | February 2021 23 


Exhibit 13. C3WP Local Capacity-Building Implementation Fidelity 


Component 1: Local Capacity Building 


Duration and breadth Use of the formative Use of C3WP 
of participation assessment tool instructional resources 
Percent of districts 
reaching district-level 76% 76% 76% 
threshold 
Program-level Ves Vas Vee 


threshold met 


We measured components 2 through 4, which focused on implementation of the key 
components of the program in each of the partner districts annually, once for the 2018—2019 
school year and once for 2019-2020 school year. In Year 1, C3WP met the overall program 
fidelity thresholds for intensity (component 2), content (component 3), and strategies 
(component 4) (Exhibit 14). In the majority of partner districts, most teachers participated in PD 
that was intensive, included sufficient content focused on argument, C3WP instructional 
resources, and the formative assessment tool, and used enactment-focused strategies. In the 
Year 2, C3WP met the overall program fidelity thresholds for content of PD (component 3) and 
strategies (component 4), but it did not meet the thresholds for intensity of PD (component 1) 
(Exhibit 15). In Year 2, a little over half the districts (57%) met the threshold for number of hours 
of participation by teachers in C3WP PD. Fewer districts (22%) met the threshold for number of 
C3WP hours over 2 years for teachers in grades 7—10. Local Writing Projects sites were 
halfway through the spring semester of Year 2 when the COVID-19 pandemic began forcing 
schools to remote learning. The health emergency impacted local Writing Projects sites’ ability 
to continue planned PD activities. Since C3WP’s secondary program was designed to last over 
2 years, we also examined to what extent teachers participated over 2 years. Overall, far fewer 
districts met this threshold. 


Exhibit 14. Year 1 C3WP District Implementation Fidelity 


Component 4: 
PD Strategies 


Sonipolenie: Component 3: Content of PD 


Intensity of PD 


Duration and Focus on Use of Use of 
: : formative Focus on 
breadth of argument instructional 
aie Be assessment enactment 
participation writing resources ies, 
Percent of 
districts reaching 82% 91% 100% 100% 100% 
district-level 
threshold 
progtam-lever Yes Yes Yes Yes Yes 


threshold met 
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Exhibit 15. Year 2 C3WP District Implementation Fidelity 


F : Component 
Suet WUC Component 3: Content of PD 4: PD 
Strategies 
Duration and Duration and Use of 
breadth of breadth of Focus on Use of formative Focus on 


argument __ instructional 


ca assessment enactment 
writing resources 


tool 


participation _ participation 
in Year 2 over 2 years 


Percent of 

districts 

reaching 57% 22% 78% 78% 100% 96% 
district-level 

threshold 


Program 
level 
threshold 
met 


No No Yes Yes Yes Yes 


To understand the extent to which early-start teachers implemented C3WP in their classrooms, 
we administered a survey at the end of years 1 and 2 asking teachers which resources they 
implemented, how often, and any modifications they made. Exhibit 16 shows the response rate 
for the grade 7—10 teacher survey, which was administered during the spring of 2019 and 2020. 
The table includes the total number of eligible teachers we sent the survey to and the number of 
survey respondents. 


Exhibit 16. Grades 7-10 Teacher Survey Response Rate (Treatment Only) 


All Teachers Respondents Response Rate 
Spring 2019 168 144 86% 
Spring 2020 160 126 79% 


Treatment teachers were expected to complete at least four C3WP cycles of instruction per year 
of implementation. Most treatment teachers (83%) reported completing the four required cycles 
of instruction in Year 1. Just 61% of teachers in Year 2 reported completing at least four cycles, 
although most teachers (93%) completed at least three when spring 2020 was interrupted by 
COVID-19. The survey also asked teachers to indicate the instructional resources they used 
during the year. In Year 1, the top four most used instructional resources focused on connecting 
evidence to claims (91%), writing and revising claims (89%), writing into the day (63%), and 
ranking evidence (48%). In Year 2, the top four most used resources focused on writing and 
revising claims (74%), connecting evidence to claims (74%), writing into the day (57%), and 
making moves with evidence (57%). 


Teachers were also expected to use Routine Argument Writing between the multi-day 
instructional resources. In Year 1, the vast majority (92%) of teachers reported using Routine 
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Argument Writing regularly or a few times throughout the year. Teachers reported using the 
Using Sources Tool at high rates overall in Year 1 (94%) and Year 2 (92%). However, fewer 
teachers reported using the Using Sources Tool four or more times, which would have aligned 
with the number of cycles of instruction expected each year. We found that 69% of teachers 
reported using the Using Sources Tool four or more times in Year 1 and 56% reported the same 
in Year 2). 


Scaling 


NWP reached the thresholds for six out of seven of the scaling indicators (Exhibit 17). NWP 
increased their national leadership capacity by increasing the number of experienced teacher 
leaders with experience providing leadership nationally to 15 and expanding the reach of C3WP 
through their website. Over the course of 10 months in 2020, the C3WP website exceed the 
50,000 minimum threshold for page views. To support scaling, NWP also focused on revising 
existing 7-10 instructional resources and creating new resources for grades 4—5. C3WP’s 
current website includes 22 standardized instructional resources for grades 7-10. NWP also 
created a set of instructional resources for grades 4—5 that represents a full arc of skill 
development from identifying arguments to writing a civic argument. 


NWP set up a certification process for sites and a micro-credentialing system for teacher 
leaders to show that sites and teachers leaders are prepared to provide C3WP PD. Seventeen 
local Writing Project sites held an C3WP Advanced Institute and completed 2 years of PD with a 
partner district. A total of 229 teacher leaders completed 30+ hours across all of the Advanced 
Institutes, exceeding the threshold of 200. NWP has yet to meet the scaling indicator for 
regional leadership: three local Writing Project sites holding regional C3WP conferences. Local 
Writing Projects sites’ ability to hold conferences was curtailed by the COVID-19 pandemic, but 
NWP reported plans for sites to hold conferences in the future. Exhibit 17 reports on the extent 
to which NWP reached the initial scale-up goals by February 2021. 


Exhibit 17. C3WP Scale-Up Measures and Thresholds by Goal 


SYor-| (=U) om efor-]| Measure Minimum Take] Cer-| Ko) g 
Threshold 
Expand NWP National Leadership Capacity 
New teacher leaders with Agendas and programs for 
experience providing national national meetings 
leadership (e.g., leading a OR 
training at an NWP national Invitations to join C3WP 412 individuals 39 
meeting) 


leadership team for individuals 
not on the leadership team 
during the i3 Validation grant 
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SYor-| (=U) om efor-]| Measure Minimum Take | Core] ve) g 


Threshold 
Users of C3WP Open Google and website analytics January 1, 2019 — 
Educational Resources for page views on updated December 31, 209: 
website, collected annually by —_59 990 page 91,315 page views 
NWP and provided to the views January 1, 2020 — 
research team December 31, 2020: 
69,477,392 page 
views 
Develop and Codify Instructional and Formative Assessment Resources 
Secondary 
Resources fo grades 7-10 Number of resources posted Resources: 22 
standardized in format and : 20 resources 
: Sun on C3WP website resources posted 
revised for accessibility : 
and standardized 
Full year arc of resources that Winberelenentar 
New resources for grades 4— shows a progressive set of 1 full set of ae 42 y 
5 argument-writing skills posted resources ; 
on C3WP Website resources posted 
Invest in Regional Leadership 
Identification of C3\WP model Local Writing Project sites that 3 local 0 
sites co-lead regional C3WP Writing 
conferences Project sites 
Develop site certification and micro-credentialing system 
Sites certified to offer C3WP ae ; : 
(through additional grant ey lihcd ae ale 16 local 
funding or through fee-for- Writing 17 sites 


provide 2 years of C3WP PD 


to partner districts projet sites 


service or other funding 


source) 

Teacher leaders who : : 229 teacher leaders 
Professional Learning 200 teacher 

complete 30+ hours of , completed at least 
Tracking Form leaders 


Advanced Institute 30 hours 


Impacts on Student Writing 


Exhibit 18 provides descriptive statistics for student outcome data by treatment status. 


Exhibit 18. Year 1 Student Prompt Outcomes by AWC-SBA Components 


Treatment Foy al age) oN -1 | 
Outcome Mean SD Districtn Studentn Mean SD Districtn Studentn mean SD 
Content 2.85 1.14 23 932 2.65 1.12 24 924 2.75 1.14 
Stance 2.88 1.17 23 932 2.69 1.14 24 924 2.79 1.16 
Structure 2.79 1.14 23 932 2.61 1.12 24 924 2.70 1.13 
Conventions 2.91 1.18 23 932 2.70 1.18 24 924 2.81 1.18 


Exhibit 19 shows the estimated effect of 1 year of C3WP on grade 7-9 students’ source-based 
argument writing, as measured by the four attributes measured on the AWC-SBA. The exhibit 

presents the estimated treatment point estimate and standard error, sample sizes, p-value, and 
significance of treatment effect of the model using standard values. Effect size is calculated as 
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Hedges’ g (i.e., the covariate-adjusted mean difference minus the treatment coefficient divided 
by the unadjusted pooled within-group SD). We estimate that C3WP has a positive and 
statistically significant impact on all four attributes of writing measured. 


Exhibit 19. Estimated Impacts of 1 Year of C3WP on Grades 7—9 Students’ Source-Based 
Argument Writing, as Measured by the AWC-SBA 


Meets 

Estimated Adjusted Critical 
Treatment Student District Effect Critical fe 

OV] Kore) tay :) Effect i] n n i Size p-value value? 
Content 0.19 0.04 1,856 47 0.00 *** 0.19 .025 Yes 
Stance 0.19 0.04 1,856 47 0.00 *** 0.19 .050 Yes 
Structure 0.17 0.04 1,856 47 0.00 *** 0.18 017 Yes 
Conventions 0.19 0.04 1,856 47 0.00 *** 0.19 n/a n/a 


Note: *** p < .001. SE stands for “standard error” and n/a for “not applicable.” Adjusted critical p-values represent the 
critical p-value to assess the estimate given the Benjamini-Hochberg correction for multiple comparisons within the 
same domain. 


Summary and Discussion 


Overall, we find that C3WP was largely implemented with fidelity. Across the 2017-18 and 
2018-19 school years, over 75% of local Writing Project sites met fidelity goals to develop local 
capacity by training teacher leaders to support program implementation in treatment districts. In 
2018-19 (study Year 1), nearly all treatment districts received the target amounts, foci, and 
strategies for implementation. In 2019-20, only 57% of treatment districts received the number 
of hours of intended PD, most likely due to COVID-19-related school closures. Districts received 
PD consistent with the intended content and strategies. As a result of C3WP, students 
outscored control students on a complex performance task: reading multiple non-fiction sources 
and writing an argument drawing on these sources. This performance task is well-aligned to 
college- and career-ready state standards and similar to performance tasks on many state 
assessments. 


This is the third rigorous impact study demonstrating a positive impact of C3WP on secondary 
students’ argument-writing outcomes. The size of the estimated impacts are similar in size and 
sometimes larger than those observed in the initial i3 Validation study (Gallagher et. al, 2017). 
Notably, however, the earlier estimates were based on 2 years of program implementation: 1 
year when sites had substantial discretion about the PD approach, and 1 year when all sites 
implemented programs similar to C3WP as it was implemented in this study. 
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